feat: Provide support for batch scraping #2459
Conversation
|
Thank you for your contribution! 🙏 We will review it as soon as possible. |
src/Promitor.Core.Scraping/Configuration/Model/Metrics/ScrapeDefinitionBatch.cs
Outdated
Show resolved
Hide resolved
|
Thanks! Is it safe to say that the top of the diagram is current and bottom shows these changes? |
tomkerkhove
changed the title
src/Promitor.Core.Scraping/Configuration/Model/AzureMetricConfiguration.cs
Show resolved
Hide resolved
src/Promitor.Core.Scraping/Configuration/Model/MetricBatchScrapeConfig.cs
Outdated
Show resolved
Hide resolved
src/Promitor.Core.Scraping/Configuration/Model/Metrics/BatchScrapeDefinition.cs
Outdated
Show resolved
Hide resolved
src/Promitor.Core.Scraping/Configuration/Model/Metrics/BatchScrapeDefinition.cs
Outdated
Show resolved
Hide resolved
src/Promitor.Core.Scraping/Configuration/Model/Metrics/ScrapeDefinition.cs
Outdated
Show resolved
Hide resolved
src/Promitor.Core.Scraping/Configuration/Model/Metrics/ScrapeDefinitionBatch.cs
Outdated
Show resolved
Hide resolved
src/Promitor.Core.Scraping/Configuration/Model/Metrics/ScrapeDefinitionBatch.cs
Outdated
Show resolved
Hide resolved
|
Hey I'm revisiting this PR now since SDK migration is almost across the finish line. Do you know if the batch scraping API is available in sovereign clouds and what URLs we should use? We are particularly interested in the US Gov cloud. I can't find any information online(Azure doesn't really mentioned the existence of sovereign clouds to begin with) |
I do not know, sorry! What is the URL for public cloud? |
|
|
Awesome! How did you find this out? |
|
I reached out to the team :D |
|
@hkfgo Do you know if you'll have time to finalize this PR next week? I'd like to cut a release next week. |
We should plan for batch scraping to be on the release after this coming one. There are still a decent amount of work left and I'm not able to work on this full time. I do have time to get SDK migration done though. In that case we should have something major-ish to release :) |
|
That's OK - Thank you! |
|
@hkfgo Any thoughts on this PR? Are you able to continue working on it? |
Hey I got caught up in some other work and took a few weeks PTO. I'm back working on it now :) |
|
Any thoughts on this btw: #2459 (comment)? |
|
Great! We're ready to go, let me know when the CI is passing 👌 |
I think this is okay to merge as is, since we are pretty sure now that the reason for failure is test flakiness. The only failing part of CI are the Scraper OTEL Collector tests on Linux(WIndows image built just fine). I don't mind retrying every now and then but might be better to fix the flakiness through a separate PR. Wdyt? |
|
Oh hey the CI's passed after a few retries |
|
Ugh sorry I just realized I hadn't resolved all comments. Should be good to merge now |
|
Thanks a ton for all this hard work! |
This reverts commit 5fa547b. Signed-off-by: Tom Kerkhove <[email protected]>
Hey Tom, hope all is well!
I've attached this diagram to illustrate my proposed change.
How to Batch?
Scrape jobs are currently defined per <resource, metric>. I'm taking the simplest approach to just batch resources looking for the same metric.
Max batch size will be a configurable parameter, under runtime configuration.
Batch Job Execution Flow
Changes to Internal Metrics
Internal Promitor metrics will need to be modified under batch scraping mode.
promitor_scrape_successandpromitor_scrape_errormetrics should be appended with an additional tagbatched, to indicate the mode of operation(0/1)A histogram metric
promitor_batch_sizeshould be added to track actual batch sizespromitor_ratelimit_arm_throttled/promitor_ratelimit_armmetrics will no longer be published, since batch scraping does not go through ARM.Other considerations
Batch scraping will not be free anymore but comes with other benefits(like no longer being subject to ARM throttling). This should be documented for all users
Implements #2284